Decentralized Policy Gradient Descent Ascent for Safe Multi-Agent Reinforcement Learning

This paper deals with distributed reinforcement learning problems safety constraints. In particular, we consider that a team of agents cooperate in shared environment, where each agent has its individual reward function and constraints involve all agents' joint actions. As such, the aim to maximize team-average long-term return, subject More intriguingly, no central controller is assumed coordinate agents, both rewards are only known locally/privately. Instead, connected by peer-to-peer communication network share information their neighbors. this work, first formulate problem as constrained Markov decision process (D-CMDP) networked agents. Then, propose decentralized policy gradient (PG) method, Safe Dec-PG, perform optimization based on D-CMDP model over network. Convergence guarantees, together numerical results, showcase superiority proposed algorithm. To best our knowledge, PG algorithm accounts for coupled quantifiable convergence rate multi-agent learning. Finally, emphasize also novel solving class stochastic nonconvex-concave minimax problems, design corresponding theoretical analysis independent interest.